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, Abstract 

' Given a Banach space X and one of its compact sets we consider the problem 

of finding a good n dimensional space Xn C X which can be used to approximate the 
elements of J-. The best possible error we can achieve for such an approximation is 
' given by the Kolmogorov width dn{J^)x- However, finding the space which gives this 

performance is typically numerically intractable. Recently, a new greedy strategy for 
', obtaining good spaces was given in the context of the reduced basis method for solving 

I a parametric family of PDEs. The performance of this greedy algorithm was initially 

analyzed in [2] in the case X = "H is a Hilbert space. The results of [2] were significantly 
improved on in [T]. The purpose of the present paper is to give a new analysis of the 
performance of such greedy algorithms. Our analysis not only gives improved results 
^ ■ for the Hilbert space case but can also be applied to the same greedy procedure in 

. general Banach spaces. 
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^ ; 1 Introduction 

Let X be a Banach space with norm || ■ || := || ■ \\x, and let J-" be one of its compact subsets. 
For notational convenience only, we shall assume that the elements / of J-" satisfy \\f\\x < 1- 
We consider the following greedy algorithm for generating approximation spaces for J-". We 
first choose a function /o such that 

ll/o||=mc^||/||. (1.1) 
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Assuming {/o, . . . , /n_i} and Vn := span{/o, . . . , /„_i} have been selected, we then take 
fn^J^ such that 

dist(/„, K)^!! = maxdist(/, (1-2) 

and define 

an := (TniJ^)x ■■= dist(/„,K)x := sup inf (1-3) 

This greedy algorithm was introduced, for the case X is a Hilbert space, in the reduced basis 
method [S], E] for solving a family of PDEs. Certain variants of this algorithm, known as weak 
greedy algorithms, described below, are now numerically implemented with great success in 
the reduced basis method. Our interest in this paper will be in the approximation properties 
of this algorithm and its weak variant. 

We are interested in how well the space Vn approximates the elements of J-" and for this 
purpose we compare its performance with the best possible performance which is given by 
the Kolmogorov width dn{J^)x of J-" defined by 

dn := dn{J^)x ■= i^i sup dist(/, (1-4) 

where the infimum is taken over all n dimensional subspaces Y oi X. We refer the reader to 
[1] for a general discussion of Kolmogorov widths. We also define 

do := do{J^)x ■= max ||/|| = o-o(-^)x, 

which corresponds to approximating by zero dimensional spaces. 

Of course, if (o"„)„>o decays at a rate comparable to ((i„)„>o, this would mean that the 
greedy selection provides essentially the best possible accuracy attainable by n-dimensional 
subspaces. Various comparisons have been given between (j„ and dn- A first result in this 
direction, in the case that X is a Hilbert space "H, was given in [2] where it was proved that 

(TniJ'U < Cn2^dn{T)n, (1-5) 

with C an absolute constant. While this is an interesting comparison, it is only useful if 
dni^)^ decays to zero faster than ?t,~^2~". Various improvements on (11. 5p were given in 
[1], again in the Hilbert space setting. We mention two of these. It was shown that if 
dn{.J^)H < Cri"", n = 1, 2, . . . , then 

cTniJ'U < C'^n~^. (1.6) 

This shows that in the scale of polynomial decay the greedy algorithm performs with the 
same rates as n-widths. A related result was proved for sub-exponential decay. If for some 
< a < 1, we have dn{J^)n < Ce""^"", n = 1, 2, . . . , then 

a„(-F)«<C;e-^>', /3 = ^, n = l,2,.... (1.7) 

a + 1 

In numerical implementations, the greedy algorithm is too demanding since at each it- 
eration it requires finding an element from J-" which is at furthest distance from Vn- To cir- 
cumvent this difficulty, one modifies the algorithm as follows. We fix a constant < 7 < 1. 
At the first step of the algorithm, one chooses a function /q G J-" such that 

II /oil > lcroiJ^)x- 
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At the general step, if /o, . . . , fn-i have been chosen, Vn := span{/o, . . . , fn-i}, and 

c^niDx ■= dist(/,K)x, 
is the best approximation error to / from Vn we now choose j^,^ ^ such that 

(^n{fn)x > 7max(T„(/)x, (1.8) 

to be the next element in the greedy selection. Note that if 7 = 1, then the weak greedy 
algorithm reduces to the greedy algorithm that we have introduced above. 

Notice that similar to the greedy algorithm, (o"„(J-')x)n>o is also monotone decreasing. 
Of course, neither the greedy algorithm or the weak greedy algorithm give a unique sequence 
(/n)n>0; ^or is the sequence (o"„(J^)x)n>o unique. In all that follows, the notation reflects 
any sequences which can arise in the implementation of the weak greedy selection for the 
fixed value of 7. 

In the present paper, we shall first prove a lemma that we use in our new analysis of the 
weak greedy algorithm. This new analysis gives a significant improvement of the previous 
results. We mention two of these: 

The first, given in Corollary 13. 3 [ is that 

<T2n{J^)n < V2-f-Wdn{J^)n, n = l,2,.... (1.9) 

This is the first direct comparison between {cTn{J^)n)n>o and ((in(-^)w)n>o for the special case 
X is a Hilbert space "H, which guarantees a specific rate of decay for (a"„(J-')^)„>o without 
any assumption of a decay rate for {dn{J^)n)n>o- Notice that, in particular, this allows one 
to improve the sub-exponential results mentioned earlier (see Corollary 13.31) . 

The second part of our paper analyzes the performance of the greedy algorithm in a 
general Banach space. We prove estimates for the decay of (a"„(J-')x)n>o similar to those in 
the Hilbert space case, except that there is a loss of the order 0{^/n). We give examples 
which show that this loss in essence cannot be removed. However, our results for a general 
Banach space are still not definitive. For example, we have no result of the form (11. 9p because 
of the ^/n factor that appears in our Banach space results. 



2 Main lemma 

In this section, we shall prove a lemma for matrices that we employ in our analysis of weak 
greedy algorithms in both Hilbert and Banach spaces. 

Lemma 2.1 Let G = (gij) be a K x K lower triangular matrix with rows gi, . . . ,gK, W 
be any m dimensional subspace ofM.^, and P be the orthogonal projection ofR^ onto W . 
Then 

i=l I i=l ) i=l J 



where \\ ■ \\e2 is the euclidean norm of a vector in 
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Proof: We choose an orthonormal basis . . . , for the space W and complete it into 
an orthonormal basis <y9i, . . . , ip^ for M^. If we denote by $ the K x K orthogonal matrix 
whose j-th column is ipj, then the matrix C := G$ has entries Cij = (gj, ipj). We denote by 
Cj, the j-th column of C. It follows from the arithmetic geometric mean inequality for the 
numbers {HcjU^^lj^i ^^at 

m K f ^ K 



Similarly, 



, m ^ — ^ \ m ^ — ' ^ — ' m 

j=i j=i ) \ j=i 1=1 ) \ i=\ 



j=m+l \_ j=m+l J \ i=l ) 

Now, Hadamard's inequality for the matrix C and relations (12. 2 p and (12. 3p result in 

(deter < niM?.£ irE 11^8.111 \i7^Y.\\s--P&<\\l\ P'^) 



, m ^-^ - \ K — m 

j=l K i=l J K. i=l 

K 



The latter inequality and the fact that det G = Y\ 9i,i and | det C| = | det G\ gives (12.11) . □ 



1=1 



3 A new analysis for the weak greedy algorithm in a 
Hilbert space 

The purpose of this section is to obtain new results for the performance of the weak greedy 
algorithm in a Hilbert space that considerably improve on the analysis in [2] and [1]. This 
will be accomplished by making a finer comparison between (o"n(-^)w)r!.>o and {dn{J-')n)n>o 
than those given in [1]. We assume throughout this section that X = "H is a Hilbert space 
and follow the notation from pp. 

Note that in general, the weak greedy algorithm does not terminate and we obtain an 
infinite sequence /o, /i, /2, . . . . In order to have a consistent notation in what follows, we shall 
define fm := 0, m > N, if the algorithm terminates at A^, i.e. if aN{J^)n = 0. By (/*)„,>o we 
denote the orthonormal system obtained from (/n)n>o by Gram-Schmidt orthogonalization. 
It follows that the orthogonal projector P„ from "H onto Vn is given by 

n-l 

and, in particular, 

n 

fn = Pn+lfn = '^^'^njfj, 0,n,j = {fn, fj), j < n. 
j=0 
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There is no loss of generality in assuming that the infinite dimensional Hilbert space "H is 
£2(N U {0}) and that f* = Cj, where ej is the vector with a one in the coordinate indexed 
by j and is zero in all other coordinates, i.e. {ej)i = 6j^i. We adhere to this assumption 
throughout this section of the paper. 

We consider the lower triangular matrix 

^ := («ij)i^=0' ■= 0' j > ^■ 

This matrix incorporates all the information about the weak greedy algorithm on J-'. The 
following two properties characterize any lower triangular matrix A generated by such a 
greedy algorithm. With the notation := an{J^)H^ we have: 

PI: The diagonal elements of A satisfy •yan < \an,n\ < o'n- 

P2: For every m > n, one has Yl^Ln^mj — ^n- 

Indeed, PI follows from 

'^n.n ~ II /"II l|-^»i/"ll ~ II/" -Pri/n|| ; 

combined with the weak greedy selection property (11. 8p . To see P2, we note that for m > n, 

m 

E = II/™ - P-frnW < max 11/ - P„/f = al 

Remark 3.1 I] A is any matrix satisfying PI and P2 with (cr„)n>o decreasing sequence 
that converges to 0, then the rows of A form a compact subset o/£2(NU {0}). If is the set 
consisting of these rows, then one of the possible realizations of the weak greedy algorithm 
with constant 7 will choose the rows in that order and A will be the resulting matrix. 

The matrix representation A of the weak greedy algorithm was the basis of the analysis 
given in [T] and will also be critical in the proof of the next theorem. 

Theorem 3.2 For the weak greedy algorithm with constant in a Hilbert space H and for 
any compact set T , we have the following inequalities between On '■= an{J^)n o.nd dn : = 
dn{J-')n! for any N > 0, K > 1, and 1 < m < K , 

K 

1=1 

Proof: We consider the K x K matrix G = (gij) which is formed by the rows and 
columns of A with indices from {A^ + 1, . . . , + K}. Each row gj is the restriction of fN+i 
to the coordinates N + 1, . . . , N + K. Let "Hm be the m-dimensional Kolmogorov^bspace 
of "H for which dist(J-', "Hm) = dm- Then, dist{fN+i,'Hm) < dm, i = I, . . . K. Let W be the 
linearjpace which is the restriction of Hm to the coordinates N + 1, . . . ,N + K. In general, 
dim(Vr) < m. Let W he an m dimensional space, W C span{eAr+i, . . . ,eN+K}, such that 
W G W and P and P are the projections in IR-^ onto W and W, respectively. Clearly, 

||^gi||^2 < llgi|l^2 < ^Af+i) i = l,...,K, (3.2) 



r/n-r k r-'^m . 
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where we have used Property P2 in the last inequahty. Note that 

- PgiWh < II& - -P&llfe = dist(gi, VT) < dist{f N+uUm) <dm, i = l,...,K. (3.3) 
It follows from Property PI that 

K K 

JJ \aN+i,N+i\ (^N+i- (3.4) 

We now apply Lemma [2?T] for this G and W, and use estimates (13. 2p . (13. 3p . and (13. 4p to 
derive (13. ip . The proof is completed. □ 
We next record some special cases of Theorem 13. 2[ 



Corollary 3.3 For the weak greedy algorithm with constant^ in a Hilbert space Ti, we have 
the following: 

(i) For any compact set J-" and n > 1, we have 

n — m 

cTniJ^) < min d^" (J^). (3.5) 

l<m<n 



In particular (J2n{^) ^ V^l ^\/dn{^), n = 1,2 . . . . 

(ii) If dn{J^) < Con"", n = 1,2,..., then an{J^) < Cin"", n = 1,2..., with Ci : = 
25-+I7-2C0. 

(iii) IfdniJ^) < Coe-^o"", n = 1, 2, . . . , then a„,(J^) < V2C^-f~^e"^'''% n = l,2..., where 
ci = 2-i-2"co, 

Proof: (i) We take N = 0, K = n and any 1 < m < n in Theorem 13. 2[ use the 
monotonicity of (o"„)„>o and the fact that do < 1 to obtain 

j = l ^ J 

Since x~^{l — x)^~^ < 2 for < x < 1, we derive (13. 5p . 

(ii) It follows from the monotonicity of (crn)„>o and (13. ip for = i^T = n and any 
1 < m < n that 



Jin ^ TT J2. ^ ^ -2n / 1 J I ^2m j2n-2m 

iniJ In — ml 



j=n+l 

In the case n = 2s and m = s we have 

(Jis < v^7~Vo"2s4- (3.7) 

Now we prove our claim by contradiction. Suppose it is not true and M is the first value 
where aui,^) > CiM"°. Let us first assume M = 4s. From (13. 7p . we have 

CT4S < V27"' VCi(2s)-VCos-" = V2i-"CoCi7-^s-'^, (3.8) 
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where we have used the fact that < Ci(2s) " and ds < Cqs It follows that 



and therefore 

Ci < 23^+17^2^0 < 25"+S"^C'o, 

which is the desired contradiction. If M = 4s + g, g G {1,2,3}, then it follows from fl3.8p 
and the monotonicity of (c'"„)„>o that 

Ci2-3°s-" = Ci2-°(4s)-° < Ci(4s + g)"" < (74,+^ < (74, < V2i-"CoCi7"^s-°. 
From this, we obtain 

which is the desired contradiction in this case. This completes the proof of (ii). 
fiii) From fi), we have 



^2n+i < ^2n < v^7"'v^ < v^T'^e"^"" = v^7"'e-^«'" (3.9) 
from which (iii) easily follows. □ 

Remark 3.4 Note that one can obtain a better constant Ci in (iii) if the minimum in (13. 5p 
is computed. Namely, this gives 

^ (n-m) r- — C^'^ \ HiaX 1 } 

c^2n<v^7~^Co min e"""'""^^ = y27-^Coe \<m<n\n) \ nJ _ 

l<m<n 

Then, using the fact that x"(l — x), < x < 1, has a maximum at results in a better 
constant. 

4 Bounds for the greedy algorithm in Banach spaces 

We will now derive bounds for the performance of the weak greedy algorithm in a general 
Banach space X. In this section, we will use the abbreviation cr„ := an(J^)x and dn '■ = 
dn{J^)x- As in the Hilbert space case, we associate with the greedy procedure a lower 
triangular matrix A = {aij)°°^Q in the following way. For each j = 0, 1, . . . , we let Xj G X* 
be the linear functional of norm one that satisfies 

(i) \,{Vj) = 0, (ii) Xjif,) = dist(/,, V,)x. (4.1) 

The existence of such a functional is a simple consequence of the Hahn-Banach theorem (see 
[H Chapt. IV, Cor. 14. 13]). We let A be the matrix with entries 
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From (ii) of (??) Its diagonal elements aj^j satisfy the inequality 



< ajj = dist(/j, Vj)x = (Tj, (4.2) 
because of the weak greedy selection property f ll.Sp . Also, each entry aij satisfies 

= \^jifi)\ = \^j{fi-9)\ < \\^j\\x*\\fi- gW = Wfi-gW, j < h 

for every g &Vj, since ^jiVj) = 0. Therefore we have 

\aij\ < dist(/i, Vj) < aj, j < i. (4.3) 

Theorem 4.1 For the weak greedy algorithm with constant ^ in a Banach space X and for 
any compact set J-" contained in the unit ball of X , we have the following inequalities between 
O'n '■= 0'n{J^)x o-nd dn '■= dn{J^)x: for any N > 0, K > 1, and 1 < m < K , 

(^N+i <2 K 7 (T^^i '> d^ . (4.4) 

i=l I 1=1 J 



Proof: As in the proof of Theorem 13. 2[ we consider the K x K matrix G which is 
formed by the rows and columns of A with indices from {N + 1, . . . , + K}. Let X^ be 
the Kolmogorov subspace of X for which dist(J-', X^) = dm- For each i, there is an element 
hi G Xm such that 

II /i -hi\\= dist{fi,Xm)x < dm, 

and therefore 

|A,(/^) - XAK)\ = |A,(/, - h,)\ < \\X,\\x4f^ - h^\\ < dm. (4.5) 

We now consider the vectors {\N+i{h), . . . , Xx+Kih)), h G Xm- They span a space 
W C M.^ of dimension < m. We assume that dim(W^) = m (a slight notational adjustment 
has to be made if dim(14^) < m). It follows from f l4.5p that each row gj of G can be 
approximated by a vector from W in the i^o norm to accuracy d^, and therefore in the £2 
norm to accuracy y/Kdm.- Let P be the orthogonal projection of onto W. Hence, we 
have 

||gi - PgiWk < ^dm., i = l,...,K. (4.6) 
It also follows from f l4.3p that 

1/2 



N+j 



and therefore 

i=l i=l j=l 4=1 
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Next, we apply Lemma |2TT] for this G and W and use estimates (14. 2p . f l4.6p and (14. 7p to 
derive 



( K ^ 

< 2^i^^-™ dr-\ 



i=l 



and the proof is completed. □ 
In analogy with Corollary 13. 3^ we have the following special results for the weak greedy 
algorithm in a general Banach space. 

Corollary 4.2 Suppose that X is a Banach space. For the weak greedy algorithm with a 
constant 7, applied to a compact set T contained in the unit ball of X , the following holds 
for an ■■= cr,,{J')x and rf„ := dn{J^)x, n = 1, 2, . . . , 

(i) For any such compact set T and n > 1, we have 



2n 

n — m 



an<^l'^ min IS" o}) d™" . (4.8) 

l<m<n I ^ * I 



In particular (T21 < 2j~^\/7de, i = 1,2 ... . 

(ii) If for a > 0, we have dn < Cqu'"" , n = 1,2, . . . , then for any < /3 < min{a, 1/2}, 
we have cr„ < Cin~"~^^^'^~^^ , n = 1,2 ... , with 



Ci := max |Co4^°+^7-' V^iT ) ' „^i^^7^^" > 

(iii) If for a > 0, we have dn < CqC''^'"-" , n = 1,2, ... , then an < 1/260^7"^ A/ne"*^^"", 
n = 1,2..., where c\ = 2~^~^°co. The factor ^/n can be deleted by reducing the constant c\. 

Proof: The proofs are similar to those of Corollary 13.31 except that we use (14. 4p in place 

(i) We take X = ^, K = n, and any 1 < m < n in (14. 4p and use the monotonicity of 
(o-n)n>o to obtain 



m 



—In ^ r)n ^n—m „ ,—2n ) \ ^ „2 I j2n—2m r a n\ 



1=1 

If we take a 2n-th root of both sides, we arrive at (14. 8p . In particular, if n = 2£ and m 
we have 

cT2e < ^i-\2ifl\Y?l^a'lYl'^, < = 27"' V^, 

where we have used the fact that all a,- < 1. 
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(ii) It follows from the monotonicity of (cr„)„>o and (14. 4p for N = K = n and any 
1 < m < n that 

< v^7"V^4'r'^ ^■■=-- (4.10) 



n 



Given our /3, we define m =: L2|+T'^J ^ (m < for n > 2 > 2/3 + 1). It follows that 

S^'^,(Jl,,Jl^^l). (4.11) 



n + 1 2/3 + 1 n 

We next prove (ii) by contradiction. Suppose it is not true and M is the first value where 
(Jm > CiM"""*"^"*"^/^. Clearly, because of the definition of Ci, and the fact that < 1, we 
must have M > 7. We first consider the case M = 2n, and therefore n > 3. From (I4.10p we 
have 

Ci(2n)-"+''+i/2 < < y2;^7-^Cfn^(-"+'^+i/2)Co^-'^((5n)-"(i-'^), 
where we have used the fact that cr„ < Cin~°'~^^~^^^'^ and dm < Co'm~°'. It follows that 

Cl~' < Co^-^2"-^-ir"(^-^)n'^(^-^), 

and therefore 

Ci < Co2T^7-i^(5~"n' 
Since for n > 4 > 2{2(3 + 1), we have 

5 < 2'l2(3+\) ^ ^' "^""^ therefore < 2(2/3 + 1). 

This gives 

2/3 + 1 - 2«TT (2/3 + 1)^ , 2i!±i(5 (2/3+1)' of2/3+i)2 

— =^ < — —, and thus n 2 ^ 2/3+1^ < n " < 2^'''^+^^ . 

2 1 — n 



r 2/3 

a-/3 1 ^ 2/3 + 1 2?+T 



Then, for /3 < min{a, 1/2} 

:(a-/3)(2/3+l)^-2(2/3+l) 

2/3 + 1 



< Co2^(-+^- " < Co2^(-+S-^ (^) " , (4.12) 

which is the desired contradiction. Likewise, if M = 2n + 1 (since M > 7, we have n > 3), 
for —a + /3 + l/2<0 (which is the meaningful case), 

Ci2-"+^+i/2(2n)-"+^+i/2 ^ ^^^2n + l)-°+/'+V2 < ^^^^^ < 



and following the same argument as above we get 
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where we have used that 2 ^(i-i) < and the proof is completed, 
(iii) From (i), we have 

from which (iii) easily follows. □ 

5 Lower bounds in a Banach space 

It is natural to ask whether the factor ^/n is necessary when proving results in a Banach 
space. Here, we shall provide examples which show that a loss of this type is indeed necessary. 
However, as it will be seen, there is still a small gap between what we have proved for direct 
estimates and what the examples below provide. 

Let us begin by considering the space X := £oo(N U {0}) equipped with its usual norm. 
We consider a monotone decreasing sequence Xq > Xi > X2 > ■ ■ ■ of positive real numbers 
which converge to zero and define 

fj- "^j^ji j 0; I; ■ ■ ■ 5 

where ej, j = 0, 1, . . . are the usual coordinate vectors in M^'-''^^^. Let J-' := {/o, /i, • • • }• 
From the monotonicity of the Xj^s, the greedy algorithm for T in X can choose the elements 
from in order /q, /i, . . . . Hence, 

aj = (Tj{J^)x = Xj, j > 0. 

We want to give an upper bound for the Kolmogorov width of J-". For this, we shall use the 
following result (see (7.2) of Chapter 14 in [1]) on s-widths of the unit ball 6™ of in 



00 • 



ds{bT)x < C {ln(m/s)}'/' 1 < s < m/2. (5.1) 

Let us now define the sequence {xj}j>o so that in position 2^^^ < J < 2'^ — 1 it has the 
constant value 2"^^", for /c = 0, 1, . . . , where a > 1/2. It follows that, 

cr„(^)x = 0(n-"), n = l,2,.... 

We shall now bound the A^-width of J-" when = 2"+^ by constructing a good space Xj^ 
of dimension < for approximating J-". The space will be the span of a set E of at most 
N vectors. First, we place in E all of the vectors, ei, . . . , 62". Next, for each k = 1, . . . n, 
we use (15. ip to choose a basis for the space of dimension 2"'~^ whose vectors are supported 
on [2"+^, 2"+'^+^ — 1] and this space approximates each of the fj, j = 2"+'^, . . . , 2"+'^+^ — 1, 
in X to accuracy 

^^2-(n+fc)ay^2-("-'=)/2. We place these basis vectors in E. Notice that 
\xj\ < 2~^"" for j > 2^". This means that for the space Xpj- := span(£') with dimension < N 
we have 

dN{J^)x < dist(J^,Xjv)x < max (2-2"", max Co2-("+^^)"2-("-^^)/2y^l 

I l<k<n J 

= max /2-2"",C'o2-"("+i/2) . 2-'=("-^/2)y^l < (^^2-"(°+i/2)^ a > 1/2. 

l<fc<n J 
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From the monotonicity of {dn{J^)x)n>o, we obtain that 

rfn(^)x < C^2n-"-^/^ n=l,2,.... 

This example shows that the factor ^/n which appears in (ii) of Corollary 14.21 can in 
general not be removed. 
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